Mining Text for Word Senses Using Independent Component Analysis

The assumption that the problem of ambiguity in text analysis can only be solved if statistical dependencies of higher than second order are considered leads us to independent component analysis (ICA), a statistical formalism that takes higher-order dependencies into account. By assuming independence, ICA is capable of detecting a set of hidden vectors if only different linear mixtures of these vectors are observable. As a test case for ICA’s applicability to natural language processing we look at the task of word sense induction. Our starting point is that we consider the co-occurrence vector of an ambiguous word as a linear mixture of its unknown sense vectors. If corpora from different domains are available, this should give us the different linear mixtures that are required for ICA. It turns out that the independent sense vectors derived by ICA from the distributional differences of word usage reflect a word’s meanings surprisingly well.